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1 Introduction 

NIST  is  continuing  efforts  to  standardize  the  evaluation  of  coordinate  measuring  systems 
(CMS)  software  by  developing  the  Algorithm  Testing  and  Evaluation  Program  (ATEP)  [1].  This 
program  will  be  the  first  calibration  service  in  the  United  States  for  dimensional  metrology 
software.  To  test  and  evaluate  CMS  software,  ATEP  will  rely  on  three  components:  NIST’s 
Algorithm  Testing  System  (ATS),  test  procedures  based  on  national  standards  [2],  and  reference 
algorithms.  This  document  focuses  on  a description  of  a selection  process  for  standard  reference 
algorithms.  The  ATS  and  the  test  procedures  are  briefly  described  in  Section  2. 

Reference  algorithms  provide  a baseline  of  performance  for  testing  and  assessing  the  data 
analysis  software  found  in  CMSs.  Presently,  NIST-developed  geometric  fitting  algorithms  are 
incorporated  in  the  ATS.  These  algorithms  are  used  as  default  reference  algorithms  to  provide 
a performance  baseline.  Moreover,  since 
many  geometric  fitting  algorithms  could  serve 
as  reference  algorithms,  the  ATS  allows  the 
use  of  external  algorithms  as  reference  algo- 
rithms. NIST  will  address  mechanisms  for 
selecting  new  and  replacing  existing  standard 
reference  algorithms  (SRAs)  with  ones  that 
provide  an  improved  performance  baseline. 

There  will  exist  SRAs  for  different  objective 
functions.  The  SRAs  will  then  be  the  current 
baseline  for  CMS  data  analysis  software 
performance  evaluation. 

This  report  is  directed  towards  the 
dimensional  metrology  community  interested 
in  standardizing  geometric  fitting  algorithms 
used  in  CMSs  and  is  intended  to  stimulate 
suggestions  for  establishing  an  SRA  selection 
process.  This  report  describes  a procedure  for 
selecting  reference  algorithms  as  new  SRAs 
or  as  replacements  for  existing  SRAs.  An  ac- 
ceptance procedure  for  evaluating  and  selecting  the  SRAs  will  be  identified.  The  acceptance 
procedure  assures  that  evolving  objectives  for  software  and  algorithm  technologies  are  not 
overlooked.  The  SRA  selection  process  provides  control  of  variation  among  national  reference 
algorithms  and  ensures  addressing  the  needs  of  industry. 
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Figure  1 Candidate  software  goes  through  an 
alpha  test  phase  and  a beta  test  phase  before  it 
can  be  selected  as  a standard  reference  algo- 
rithm. 
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Section  2 provides  a brief  description  of  existing  problems  with  CMS  software  and 
NIST’s  involvement  in  providing  solutions.  Section  3 is  an  overview  of  the  SRA  selection  proce- 
dure. The  selection  procedure  is  envisioned  as  a two-stage  process:  an  alpha  phase  conducted 
by  NIST  and  a beta  phase  of  public  review.  Figure  1 illustrates  the  overall  picture  of  the  SRA 
selection  process.  In  Sections  4 and  5 the  details  of  the  alpha  and  beta  phases  are  described, 
respectively.  Section  6 provides  a summary  of  related  activities. 

2 Background 

CMSs  are  widely  used  by  the  dimensional  metrology  community  during  the  inspection 
of  manufactured  parts.  The  data  analysis  software  within  the  CMSs  (the  software  that  evaluates 
coordinate  data  and  produces  dimensional  measurements)  is  becoming  increasingly  important 
because  of  the  significant  effects  such  software  has  on  the  reported  measurements.  For  example, 
software  computations  used  to  convert  raw  data  to  reported  results  can  be  a major  source  of  error 
in  a measurement  system  [3].  There  are  no  standards  or  available  methods  for  testing  and 
evaluating  the  effects  of  data  analysis  software  on  the  overall  uncertainty  of  measurements  [4]. 
How  to  test  coordinate  measuring  system  software  and  how  the  measurement  uncertainty  is 
affected  is  currently  being  researched  in  the  United  States  [5],  in  Great  Britain  [6],  and  in 
Germany  [7]. 


for  testing  the  performance  of  data  analysis  software  used  in  CMSs. 


As  part  of  NIST’s  contribution  to  an  ongoing  effort  by  industry  standard  groups  to 
establish  standards  for  testing  software  used  in  inspection  systems,  NIST  developed  the  ATS. 
The  ATS  is  a software  package  for  evaluating  the  performance  of  data  analysis  software.  The 
architecture  of  the  ATS  is  shown  in  Figure  2.  The  ATS  provides  capabilities  for  comparing  the 
results  of  different  fitting  routines  to  a set  of  reference  algorithms.  The  ATS  allows  the  user  to 
generate  test  data  for  various  geometries  with  simulated  form  errors  and  simulated  measurement 
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errors;  run  test  data  through  the  reference  algorithms;  import  fits  computed  by  other  algorithms; 
and  compare  the  results  of  algorithms. 

The  NIST  ATS  is  being  used  as  the  basis  for  developing  a new  national  standard,  ASME 
B89.4.10,  Methods  for  Performance  Evaluation  of  Coordinate  Measuring  System  Software.  This 
standard  will  provide  guidelines  for  evaluating  CMS  software.  In  addition  this  standard  will 
provide  the  test  procedures  to  be  used  in  the  proposed  NIST  ATEP. 

ATEP  will  be  a NIST  service  that  combines  the  ATS,  test  procedures  based  on  national 
standards,  and  control  over  the  ATS  reference  algorithms.  NIST  will  use  these  components  to 
evaluate  the  data  analysis  software.  Figure  3 illustrates  the  basic  idea  for  the  proposed  NIST 
program  to  test  and  evaluate  CMS  data  analysis  software.  The  customer  receives  NIST-generated 
data  sets  and  generates  fit  results  with  the  data  analysis  software  to  be  tested.  NIST  generates 


Figure  3 Proposed  NIST  Algorithm  Testing  and  Evaluation  Program  for  performance  evaluation 
of  CMS  data  analysis  software. 


fit  results  from  the  same  data  sets  using  the  ATS’s  reference  algorithms.  NIST  then  compares 
the  two  sets  of  fit  results.  According  to  predefined  test  procedures,  NIST  provides  the  customer 
with  an  evaluation  of  the  software. 

The  rest  of  this  report  describes  the  SRA  selection  process,  as  currently  envisioned,  that 
NIST  will  use  to  maintain  and  update  the  reference  algorithms  used  in  the  ATEP. 
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3 Overview  of  SRA  Selection  Procedure 

Figure  4 illustrates  the  SRA  selection  process.  The  process  starts  with  the  development 
of  a new  algorithm.  The  algorithm  may  be  developed  by  NIST,  private  industry  or  academia. 
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Figure  4 The  SRA  Selection  Process.  The  process  proceeds  in  two  phases,  shown  by  the  dashec 
boxes. 

Before  a proposed  algorithm  is  submitted,  a developer  contacts  NIST.  NIST  and  the  developer 
agree  on  a statement  of  work.  The  statement  of  work  specifies  what  information  is  required  to 
accept  the  algorithm  as  a candidate  for  an  SRA.  The  developer  submits  the  proposed  algorithm 
and  the  required  information  to  NIST.  NIST  reviews  the  submission  for  completeness  of  infor- 
mation and  adequacy  of  terms  according  to  the  statement  of  work.  Additionally,  NIST 
acknowledges  receipt  of  the  request.  If  the  submission  is  incomplete  or  inadequate,  it  is  returned 
to  the  developer  with  an  explanation.  Otherwise,  NIST  begins  the  alpha  test.  After  evaluating 
the  alpha  test  results,  NIST  determines  whether  the  algorithm  is  suited  for  beta  testing.  If  the 
algorithm  fails  the  alpha  test,  it  is  returned  to  the  developer  with  an  explanation.  If  the  algorithm 
passes  the  alpha  test,  it  is  distributed  to  the  coordinate  measurement  community  for  beta  testing. 
NIST  evaluates  the  beta  test  results  and  tries  to  build  a consensus  for  the  disposition  of  the 
submitted  algorithm.  If  the  algorithm  is  not  suitable,  it  is  returned  to  the  developer.  Otherwise, 
it  becomes  a standard  reference  algorithm  and  can  be  used  to  improve  or  enhance  the  testing 
capabilities  offered  through  the  ATER 

4 Details  of  Alpha  Testing 

Before  NIST  begins  the  alpha  test,  the  developer  is  to  meet  certain  requirements.  First, 
the  developer  provides  a statement  of  benefits  and  applicability  of  the  algorithm.  The  developer 
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states  why  the  submitted  algorithm  would  be  useful  as  an  SRA.  Additionally,  if  the  algorithm 
is  a replacement  of  an  existing  SRA,  the  developer  states  why  the  submitted  algorithm  is  better. 

Second,  the  developer  provides  technical  documentation  about  the  algorithm:  a written, 
detailed  description  of  what  the  algorithm  does  and  how  it  does  it.  Pseudo-code  and  source  code 
(preferably  in  ANSI  standard  C)  are  optional.  Code  in  other  programming  languages  are 
addressed  in  the  statement  of  work.  The  documentation  should  specify  the  theoretical  basis  for 
the  algorithm.  An  objective  function,  input  and  output  considerations  (restrictions,  data  points, 
parameters),  and  error  handling  should  be  clearly  stated.  The  design  of  the  abstract  implementa- 
tion should  also  be  described.  That  is,  the  underlying  mathematical  and  numerical  details  (e.g., 
minimization  methods,  solution  methods)  should  be  documented.  The  algorithms  that  implement 
the  mathematical  or  numerical  methods  used,  or  references  to  these  algorithms  in  literature,  are 
to  be  included.  Portions  of  the  documentation  that  are  proprietary  should  be  marked  as  such. 
Documentation  that  is  not  marked  is  assumed  to  be  public  information. 

Last,  certain  non-technical  issues  will  be  addressed  before  the  alpha  test  begins.  Because 
the  process  of  selecting  an  SRA  requires  public  participation,  the  SRA  becomes  public 
information.  In  addition,  the  developer  and  NIST  will  coordinate  the  algorithm’s  implementation 
and  integration  into  the  ATS.  If  the  two  parties  agree  that  NIST  will  implement  the  proposed 
algorithm,  then  the  developer  should  be  available  for  consultation.  This  arrangement  may  be 
informal  or  formal  (e.g..  Cooperative  Research  and  Development  Agreement).  If  the  developer 
agrees  to  do  the  implementation,  NIST  will  provide  guidelines  for  implementing  an  algorithm 
in  the  ATS  environment.  Future  maintenance  of  the  implementation  would  be  the  responsibility 
of  NIST. 

Once  these  criteria  are  met  and  the  conditions  are  agreed  upon,  NIST  begins  the  alpha 
test.  As  the  first  step  of  the  alpha  test,  the  benefits  and  applicability  of  the  submitted  algorithm 
are  evaluated  by  surveying  the  coordinate  measurement  community.  If  there  is  sufficient  interest 
for  the  algorithm  within  the  community,  testing  continues.  This  survey  provides  a me ^ nanism 
for  eliminating  irrelevant  algorithms. 

In  the  second  step,  NIST  reviews  the  detailed  algorithm  description,  considering  three 
elements:  the  theoretical  basis  for  the  algorithm,  the  abstract  implementation  of  the  algorithm, 
and  ATS  environment  issues.  Questions  relevant  to  the  theoretical  basis  include:  Does  the 
algorithm  do  what  it  says  it  does?  Is  the  objective  accomplished?  Questions  regarding  the 
abstract  implementation  are:  What  are  the  strengths  and  weaknesses  of  the  underlying  numerical 
analysis  methods  outside  of  any  particular  computing  environment?  How  stable,  complete,  and 
accurate  are  these  methods?  Question  regarding  the  environment  issue  is  : Can  the  algorithm  be 
implemented  in  the  ATS? 

For  the  third  step,  NIST  and/or  the  developer  implement  and  integrate  the  proposed 
algorithm  into  the  ATS,  assuming  problems  encountered  during  the  review  are  resolved. 

Finally,  the  ATS  implementation  of  the  submitted  algorithm  is  tested  and  analyzed  using 
test  data  sets.  These  test  cases  are  yet  to  be  defined. 

All  the  results  from  each  step  of  the  alpha  test  are  evaluated.  If  any  problems 
encountered  during  the  alpha  test  cannot  be  resolved,  then  the  submitted  algorithm  is  returned 
to  the  developer. 
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5 Details  of  Beta  Testing 

If  the  submitted  algorithm  passes  the  alpha  test,  it  is  prepared  for  beta  testing.  Beta 
testing  is  done  by  interested  members  of  the  coordinate  measurement  community.  The  developer 
compiles  user  documentation  for  the  submitted  algorithm,  while  NIST  prepares  user  documenta- 
tion for  the  ATS  as  modified  with  the  new  algorithm. 

The  first  step  of  the  beta  test  is  the  compilation  of  the  beta  test  distribution  packet.  The 
distribution  packet  requires  an  IBM/PC  compatible  computer.  The  minimum  configuration  is: 
80286  or  higher  cpu 
DOS  Version  3.3  or  higher 
Hard  disk  with  at  least  2 MB  free  space 
EGA  or  VGA  graphics 

The  distribution  packet  includes  the  following  items: 

Documentation 

Detailed  algorithm  description 
ATS  implementation  information 
ATS  user  documentation 
Beta  test  guidelines 
3 1/2"  Diskette 

ATS  executable  with  submitted  algorithm  implemented 

Source  code 

Test  data  sets 

Documentation 

Install  procedure 

Information  file 

Disclaimers 

Comment  sheet/questionnaire 
Cover  letter 

NIST  then  sends  the  distribution  packet  to  participating  members  of  the  coordinate 
measurement  community.  The  community  evaluates  the  submitted  algorithm  and  returns 
comments  to  NIST.  NIST  reviews  and  summarizes  community  evaluations. 

Based  on  its  own  findings  and  community  input,  NIST  writes  a position  paper  regarding 
the  disposition  of  the  submitted  algorithm  and  tries  to  build  a consensus  among  the  community. 
For  unresolved  problems,  the  action  depends  on  the  nature  of  objection.  Levels  of  problem 
severity  will  be  determined  at  a future  time. 

6 Summary 

The  SRA  selection  procedure  is  still  under  development.  Specific  test  procedures  need 
to  be  identified  and  will  be  described  in  future  reports.  The  completion  of  the  ASME  B89.4.10 
standard  for  CMS  software  performance  evaluation  will  provide  the  guidelines  necessary  to 
determine  the  test  procedures.  The  B89.4.10  Working  Group  on  Software  Performance  for 
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Coordinate  Measurement  Systems  is  currently  working  on  completing  the  draft  national  standard. 
The  draft  standard  is  now  scheduled  to  be  available  in  1994. 

The  ASME  B89.4.10  standard  will  also  provide  the  specifications  for  testing  and 
evaluating  data  analysis  software  under  NIST’s  Algorithm  Testing  and  Evaluation  Program. 
ATEP  is  also  under  development  and  is  expected  to  be  available  for  1994. 

Version  2 of  NIST’s  ATS  is  scheduled  to  be  completed  by  mid  1994.  Version  1.1  of  the 
ATS  was  distributed  to  members  of  the  ASME  committees  for  beta  testing.  The  ATS  supports 
testing  of  least-squares  fitting  algorithms  for  six  common  geometries  found  on  manufactured 
parts.  Version  2 of  the  ATS  supports  more  geometries  and  other  methods  of  geometric  fitting 
(e.g.  minimum  circumscribed).  Version  2 is  also  easier  to  use  and  provides  greater  testing  and 
evaluation  capabilities. 
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